add design doc for auto pass through hashagg #9295
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: guo-shaoge <[email protected]>
Signed-off-by: guo-shaoge <[email protected]>
ti-chi-bot
bot
added
do-not-merge/needs-linked-issue
release-note-none
Signed-off-by: guo-shaoge <[email protected]>
Signed-off-by: guo-shaoge <[email protected]>
yibin87
reviewed
Aug 12, 2024
| ## Introduction | ||
| The HashAgg pushed down to TiFlash can be a one-stage, two-stage, or three-stage. For two-stage and three-stage aggregations, the 1st hashagg is used for pre-aggregation to reduce the amount of data that needs to be shuffled. | ||
|
|
||
| However, the optimizer cannot always choose the most suitable plan based on statistics. For example, it is common to encounter cases where two-stage HashAgg is used for datasets with high NDV, resulting in poor pre-aggregation effects in the 1st hashagg. |
There was a problem hiding this comment.
1st hashagg => 1st stage hashagg?
Since, previously, we use two-stage, three-stage.
| ### Core State Switch | ||
| The 1st hashagg will follow the state transition diagram below, with each state having the following meanings: | ||
| 1. `Init`: The initial state, where it remains as long as the HashMap is smaller than a specific value(to make sure the HashMap can fit in the L2 cache). In this state, the incoming Block is inserted into the HashMap for pre-aggregation. | ||
| 2. `Adjust`: In this state, the Block is inserted into the HashMap while probing and recording the degree of aggregation for the Block. It will switch to `PreAgg` or `PassThrough`. |
There was a problem hiding this comment.
It will switch to '' , '', or 'Selective'?
| 2. `Adjust`: In this state, the Block is inserted into the HashMap while probing and recording the degree of aggregation for the Block. It will switch to `PreAgg` or `PassThrough`. | ||
| 3. `PreAgg`: In this state, the Block is inserted into the HashMap. This state lasts for N rows(see code for N) before switching back to the `Adjust` state. | ||
| 4. `PassThrough`: In this state, the Block is directly put into the memory buffer. This state lasts for M rows(see code for M) before switching back to the `Adjust` state. | ||
| 5. `Selective`: For rows which can hit the HashMap, the aggregation function is calculated directly. For rows can't hit, they are put into the pass through buffer. So in this state, the HashMap does not grow. |
There was a problem hiding this comment.
Add change back to Adjust explanation?
zanmato1984
requested changes
Aug 12, 2024
| * [Unresolved Questions](#unresolved-questions) | ||
|
|
||
| ## Introduction | ||
| The HashAgg pushed down to TiFlash can be a one-stage, two-stage, or three-stage. For two-stage and three-stage aggregations, the 1st hashagg is used for pre-aggregation to reduce the amount of data that needs to be shuffled. |
There was a problem hiding this comment.
Suggested change
| The HashAgg pushed down to TiFlash can be a one-stage, two-stage, or three-stage. For two-stage and three-stage aggregations, the 1st hashagg is used for pre-aggregation to reduce the amount of data that needs to be shuffled. | |
| The HashAgg pushed down to TiFlash can be a plan of one-stage, two-stage, or three-stage. For two-stage and three-stage aggregations, the 1st hashagg is used for pre-aggregation to reduce the amount of data that needs to be shuffled. |
There was a problem hiding this comment.
I guess you are using "hashagg" (all lower-cased) to differentiate the plan level term "HashAgg" (camel-cased). If I'm guessing right, you may just use "aggregation" because "hashagg" is not a valid word.
There was a problem hiding this comment.
done. All hashagg changed to aggregation
Signed-off-by: guo-shaoge <[email protected]>
ti-chi-bot
bot
added
needs-1-more-lgtm
zanmato1984
approved these changes
Aug 12, 2024
| This functionality is accomplished through the `Block.info.selective` array. The DAGResponseWriter will use this array to determine which rows in a Block can be sent directly and which need to be ignored. | ||
|
|
||
| ### Spill | ||
| If AutoPassThroughHashAgg is used, spilling will not occur. Once the HashMap grows large enough to require spilling, it will immediately trigger a forced pass-through(meaning all subsequent Blocks will be forced to pass through). |
There was a problem hiding this comment.
Suggested change
| If AutoPassThroughHashAgg is used, spilling will not occur. Once the HashMap grows large enough to require spilling, it will immediately trigger a forced pass-through(meaning all subsequent Blocks will be forced to pass through). | |
| If `AutoPassThroughHashAgg` is used, spilling will not occur. Once the HashMap grows large enough to require spilling, it will immediately trigger a forced pass-through(meaning all subsequent Blocks will be forced to pass through). |
| Additionally, the HashMap's Blocks will be prioritized for returning to the parent operator in order to quickly reduce memory pressure. | ||
|
|
||
| ## Impacts & Risks | ||
| Since the algorithm judges the NDV of the overall dataset based on a small amount of data, it may lead to incorrect NDV estimation for some datasets, resulting in performance regression. In the future, this issue can be mitigated by introducing algorithms like LogLogCounting. |
There was a problem hiding this comment.
Suggested change
| Since the algorithm judges the NDV of the overall dataset based on a small amount of data, it may lead to incorrect NDV estimation for some datasets, resulting in performance regression. In the future, this issue can be mitigated by introducing algorithms like LogLogCounting. | |
| Since the algorithm estimates the NDV of the overall dataset based on a small amount of data, it may lead to incorrect NDV estimation for some datasets, resulting in performance regression. In the future, this issue can be mitigated by introducing algorithms like LogLogCounting. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: yibin87, zanmato1984 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
ti-chi-bot
bot
added
lgtm
and removed
needs-1-more-lgtm
[LGTM Timeline notifier]Timeline:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #9296
Problem Summary:
What is changed and how it works?
Check List
Tests
Side effects
Documentation
Release note